Papers with information theory
Exploring efficient zero-shot synthetic dataset generation for Information Retrieval (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in large language models offer a new avenue of generating synthetic training data to train neural retrieval models for unlabelled data collections. |
| Approach: | They propose a method to generate high-quality synthetic datasets using a small language model and a filtering mechanism to ensure the quality of generated questions. |
| Outcome: | The proposed method outperforms unsupervised retrieval methods such as BM25 and pretrained monoT5. |
Q-Tuning: Queue-based Prompt Tuning for Lifelong Few-shot Language Learning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for continual prompt tuning are limited by the ever-growing parameter scale of modern language models (e.g., GPT-4 that may have 1.76 trillion parameters). |
| Approach: | They propose a method for continual prompt tuning that enables the lifelong learning of a pre-trained language model by adding a task-specific prompt to a queue of older tasks. |
| Outcome: | The proposed method outperforms the state-of-the-art methods substantially on continual prompt tuning benchmarks. |
SMURF: SeMantic and linguistic UndeRstanding Fusion for Caption Evaluation via Typicality Analysis (2021.acl-long)
Copied to clipboard
| Challenge: | Visual captioning is an open-ended area for evaluation, requiring specialized training to improve human-correlation. |
| Approach: | They propose a new evaluation framework rooted in information theory . they propose metric SPURTS and metric SMURF to measure fluency . |
| Outcome: | The proposed metrics achieve state-of-the-art correlation with human judgment compared with other evaluation metrics. |
Where Do LLMs Compose Meaning? A Layerwise Analysis of Compositional Robustness (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches to linguistic compositional processing are limited by hallucinations, limited interpretability and a lack of semantic transparency. |
| Approach: | They propose a method that pooles token representations into linguistic constituents at various layers to improve model reliability and interpretability. |
| Outcome: | Experiments on inverse definition modelling, hypernym and synonym prediction show that semantic composition is not localised to specific layers but distributed across network depth. |
Context-aware Information-theoretic Causal De-biasing for Interactive Sequence Labeling (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing deep learning models for sequence labeling are expensive and time-consuming. |
| Approach: | They propose an interactive sequence labeling that allows training directly with the user feedback . they identify context and feedback biases by formulating interactive sequence labels via a Structural Causal Model. |
| Outcome: | The proposed approach can effectively alleviate the biases and can be learnt with the user feedback. |
SH2: Self-Highlighted Hesitation Helps You Decode More Truthfully (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have made great progress in text generation but suffer from hallucinations during reasoning and generation. |
| Approach: | They propose an inference-time method to help LLMs decode truthfully by selecting tokens with the lowest probabilities and concatenating them to the original context. |
| Outcome: | The proposed method improves LLaMA-7b, LLama2-7b and Mistral-7b on hallucination tasks. |
Measuring Human Contribution in AI-Assisted Content Generation (2026.acl-long)
Copied to clipboard
Yueqi Xie, Tao Qi, Jingwei Yi, Xiyuan Yang, Ryan Whalen, Junming Huang, Qian Ding, Yu Xie, Xing Xie, Fangzhao Wu
| Challenge: | generative AI has created a new way to generate content with humans . varying degrees of human contribution in content generation poses significant challenges for the delineation of originality . |
| Approach: | They propose a framework to measure human contribution in AI-assisted content generation by calculating mutual information between human input and AI-aided output relative to self-information of AI-assist output. |
| Outcome: | The proposed measure discriminates between varying degrees of human contribution across multiple creative domains and is validated in real-world applications. |
Modeling Content Importance for Summarization with Pre-trained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on content importance do not consider semantics and context when evaluating importance. |
| Approach: | They apply information theory to pre-trained language models to define the concept of importance from the perspective of information amount. |
| Outcome: | Experiments on CNN/Daily Mail and New York Times show that the proposed model can model the importance of content better than previous methods based on F1 and ROUGE scores. |
Extending Context Window of Large Language Models via Semantic Compression (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models rely on a quadratic computation to generate long texts . current models impose limitations on the length of text inputs . |
| Approach: | They propose a semantic compression method that extends the context window of large language models . the method reduces the semantic redundancy of long inputs before passing them to the LLMs . |
| Outcome: | The proposed method extends the context window of large language models across tasks . it exhibits consistent fluency in text generation while reducing associated computational overhead. |
Stylistic Chinese Poetry Generation via Unsupervised Style Disentanglement (D18-1)
Copied to clipboard
| Challenge: | Automatic Chinese poetry generation is one of the first attempts towards computer writing. |
| Approach: | They propose a model which requires no supervised style labeling to generate stylistic poems . they incorporate mutual information, a concept in information theory, into modeling . |
| Outcome: | The proposed model generates stylistic poems without losing fluency and coherency . it is based on mutual information, a concept in information theory . |
A Grounded Typology of Word Classes (2025.naacl-long)
Copied to clipboard
| Challenge: | Using captioned images, we can quantify language function and semantics using a grounded typology approach . linguistic typology is the study of patterns and variation across the world's languages . |
| Approach: | They propose a grounded typology approach that uses images captioned across languages to quantify meaning and semantics. |
| Outcome: | The proposed approach can quantify language function and semantics using images captioned across languages. |
Vocabulary Learning via Optimal Transport for Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Empirical results show that VOLT beats widely-used vocabularies in diverse scenarios, including WMT-14 English-German translation, TED bilingual translation, and TED multilingual translation. |
| Approach: | They propose a token dictionary solution that can be used without trial training to find the best dictionary with a proper size. |
| Outcome: | The proposed solution beats widely-used vocabularies in English-German translation, TED bilingual translation, and TED multilingual translation. |
The Tug of War Within: Mitigating the Fairness-Privacy Conflicts in Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods to enhance an LLM's privacy awareness with thousands of samples decrease its fairness awareness. |
| Approach: | They propose a training-free method to Suppress the Privacy and faIrness coupled Neurons (SPIN) which theoretically and empirically decreases the mutual information between fairness and privacy awareness. |
| Outcome: | The proposed method reduces the mutual information between fairness and privacy awareness without compromising general capabilities. |
Improving Disentangled Text Representation Learning with Information-Theoretic Guidance (2020.acl-main)
Copied to clipboard
Pengyu Cheng, Martin Renqiang Min, Dinghan Shen, Christopher Malon, Yizhe Zhang, Yitong Li, Lawrence Carin
| Challenge: | Disentangled representation learning (DRL) maps different aspects of data into distinct and independent low-dimensional latent vector spaces. |
| Approach: | They propose a method that manifests disentangled representations of text without supervision on semantics by minimizing the upper bound between style and content. |
| Outcome: | The proposed method improves on conditional text generation and text-style transfer tasks and improves style preservation. |
Speculative Sampling via Exponential Races (2025.findings-acl)
Copied to clipboard
| Challenge: | Speculative decoding accelerates large language model inference using a smaller draft model. |
| Approach: | They propose a speculative decoding method that generates multiple draft tokens for each model evaluation using a more efficient draft model. |
| Outcome: | The proposed method matches state-of-the-art performance and is based on exponential races. |
Using Information Theory to Characterize Prosodic Typology: The Case of Tone, Pitch-Accent and Stress-Accent (2025.acl-long)
Copied to clipboard
| Challenge: | lexical identity and prosody are well-studied parameters of linguistic variation, but they are difficult to predict in tonal languages. |
| Approach: | They propose to characterize the relationship between lexical identity and prosody using information theory to estimate mutual information between the text and pitch curves. |
| Outcome: | The proposed hypothesis supports perspectives that view linguistic typology as gradient, rather than categorical. |
Evaluating Sequence Labeling on the basis of Information Theory (2025.acl-long)
Copied to clipboard
| Challenge: | Existing metric families focus on certain aspects of sequence labeling tasks. |
| Approach: | They propose a metric that measures how much information each token contributes depending on different aspects of the sequence. |
| Outcome: | The proposed metric can satisfy all properties simultaneously. |
Valid Necessary: Diagnosing Latent Inefficiency in Chain-of-Thought (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing reasoning step evaluators fail to distinguish “valid but inefficient” reasoning steps from necessary reasoning. |
| Approach: | They propose a training-free metric that identifies low-utility steps and a post-hoc compression strategy to quantify their impact on token usage. |
| Outcome: | The proposed metric reduces token consumption by 31–53% while maintaining accuracy at substantially higher compression rates. |
An Information-Theoretic Foundation for the Subregular Hierarchy (2026.acl-long)
Copied to clipboard
Mai Phan Quoc Hung, Khanh Nguyen Quoc, Đoàn Minh Luong, Duong Thu Ngan, Duong Thi Phuong Thao, Tuan Do
| Challenge: | Subregular theory posits that phonological patterns in natural languages occupy restricted region of formal hierarchy . phonology patterns in SL, SP, and Tier-based Strictly Local (TSL) languages are restricted . experimental evidence demonstrates humans fail to learn patterns outside these subregulate classes . |
| Approach: | They propose a subregular hypothesis that phonological patterns in natural languages occupy a restricted region of the formal language hierarchy. |
| Outcome: | The proposed framework offers a framework for understanding computational restrictions on natural language phonology. |